PARADISE: A Framework for Evaluating Spoken Dialogue Agents
نویسندگان
چکیده
This paper presents PARADISE (PARAdigm for Dialogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normalizing for task complexity.
منابع مشابه
Evaluating spoken dialogue agents with PARADISE: Two case studies
This paper presents PARADISE PARAdigm for DIalogue Sys tem Evaluation a general framework for evaluating and comparing the performance of spoken dialogue agents The framework decou ples task requirements from an agent s dialogue behaviors supports comparisons among dialogue strategies enables the calculation of per formance over subdialogues and whole dialogues speci es the relative contributio...
متن کاملEvaluating Spoken Language Systems
Spoken language systems (SLSs) for accessing information sources or services through the telephone network and the Internet are currently being trialed and deployed for a variety of tasks. Evaluating the usability of different interface designs requires a method for comparing performance of different versions of the SLS. Recently, Walker et al (1997) proposed PARADISE (PARAdigm for DIalogue Sys...
متن کاملEvaluating Dialogue Strategies in a Spoken Dialogue System for Email
This paper presents an evaluation of directed dialogue (DD) and mixed initiative (MI) strategies in a spoken language system for Email. We compare the DD strategy, in which the system controls the dialog, to the MI strategy, in which users can flexibly control the dialog. For evaluating both strategies we used the PARADISE framework, which supports comparisons among dialogue strategies. Our exp...
متن کاملEvaluating Interactive Dialogue Systems: Extending Component Evaluation to Integrated System Evaluation
This paper discusses the range of ways in which spoken dialogue system components have been evaluated and discusses approaches to evaluation that attempt to integrate component evaluation into an overall view of system performance. We will argue that the PARADISE (PARAdigm for Dialogue System Evaluation) framework has several advantages over other proposals.
متن کاملEvaluation of a Dialogue System in an Automotive Environment
In this paper we discuss features to enhance the usability of a spoken dialogue system (SDS) in an automotive environment. We describe the tests that were performed to evaluate those features, and the methods used to assess the test results. One of these methods is a modification of PARADISE, a framework for evaluating the performance of SDSs (Walker et al., 1998). We discuss its drawbacks for ...
متن کامل